Super-resolution can enhance the resolution in images and video sequences. The specific characteristic of super-resolution is that it is able to create high resolution frames which have high spatial frequencies not present in each low resolution input frame.
There are several possible strategies to increase the (overall) resolution per image: (a) adding of synthetic detail signals, e.g. to sharpen edges (LTI, see e.g. H. Schröder, H. Elsler, and M. Fritsch, “Nonlinear Picture Enhancement Techniques for Vertically Interpolated TV-Signals”, EUSIPCO Conf. Proceedings, pp. 841-844, 1986) (b) on-line or off-line optimization utilizing image models as e.g. described in U.S. Pat. No. 6,323,905 to create edges with higher steepness and less staircase artifacts and (c) reconstruction of high frequency information by using alias and sub-pixel motion between consecutive frames as e.g. described in S. Borman and R. Stevenson, “Simultaneous Multi-frame MAP Super-Resolution Video Enhancement using Spatio-temporal Priors, IEEE Int. Conference on Image Processing, 1999. All these procedures incorporate the generation of most likely information based on previously made assumptions.
A challenging task is to check the validity of these assumptions and qualify the given input video and to separate false information caused by artifacts from novel image content. Especially when multiple input frames are utilized, robust motion estimation to align the consecutive frames to one (anchor) frame is required. Mainly all known methods (e.g. as described in S. Farsiu, M. Elad, and P. Milanfar, “Video-to-Video Dynamic Superresolution for Grayscale and Color Sequences”, EURASIP Journal of Applied Signal Processing, Special Issue on Superresolution Imaging, Vol. 2006) rely on robust global motion models, but if this estimation fails or the input sequence has e.g. motion of several objects, severe artifacts will be present in the output video.
A known method for image enhancement is back-projection super-resolution as e.g. described in S. C. Park, M. K. Park, and M. G. Kang, “Super-Resolution Image Reconstruction: A Technical Overview”, IEEE Signal Processing Magazine, Vol. 20, No. 3, May 2003, pp. 21-36. Back-projection super-resolution obtains high frequency information via an iterative process. In this algorithm the degradation process between high resolution and low resolution sequence is modelled by motion compensation, blurring, and down-sampling. Then, in an iteration loop the current (received) low resolution images are compared to (modelled) low resolution frames obtained by applying the degradation process to the actual high resolution image. The difference between modelled and received images is utilized to update the current high resolution image. This is done until convergence. The first guess for the high resolution image can be computed by e.g. standard interpolation techniques.
Another known method for image enhancement is maximum-a-posteriori super resolution as e.g. described in S. Borman and R. Stevenson, “Simultaneous Multi-frame MAP Super-Resolution Video Enhancement using Spatio-temporal Priors, IEEE Int. Conference on Image Processing, 1999. Maximum-a-posteriori (MAP) super-resolution has an additional image model but otherwise a strong similarity to back-projection. Thus, the disadvantages are also the same, a required large number of input frames and a high computational load due to a large number of iterations per frame.
Still another known method for image enhancement is Kalman filter based super-resolution as e.g. described in S. Farsiu, M. Elad, and P. Milanfar, “Video-to-Video Dynamic Superresolution for Grayscale and Color Sequences”, EURASIP Journal of Applied Signal Processing, Special Issue on Superresolution Imaging, Vol. 2006. Kalman filter based super-resolution utilizes Kalman theory to compute the super-resolution frames. This algorithm also uses a temporal feed-back loop.
Another known method is described in US 2009/0245375 A1. This method has an integrated artifact reduction in the super-resolution process. For computing the current high resolution output frame, the following input signals (frames) must be available: current and next low resolution input frames, previous detail signal added to the input to obtain the previous high resolution frame and an initial high resolution frame. Then, four separate main processing blocks are utilized to obtain the output signals for the next frame: 1. Artifact suppression in the output high resolution frame by masking the current detail signal based on motion vectors and the previous detail signal. 2. Computing the next initial high resolution frame by a weighting between current motion compensated high resolution frame and next low resolution frame. 3. Detail signal generation for the current frame based on filtering, weighting of initial current high resolution frame and current low resolution frame. 4. Adding the weighted current detail signal to the current initial high resolution frame to obtain the current high resolution output frame.